System and method for estimating a direction of motion blur in an image

ABSTRACT

A method for estimating a blur direction ( 20 ) of motion blur ( 16 ) in a blurred image ( 14 ) includes the steps of (i) blurring the blurred image ( 14 ) along a first test direction ( 360 A) to create an artificially blurred first test image ( 362 A); (ii) blurring the blurred image ( 14 ) along a first perpendicular test direction ( 364 A) to create an artificially blurred first perpendicular test image ( 366 A), the first perpendicular test direction ( 366 A) being substantially perpendicular to the first test direction ( 362 A); (iii) comparing the first test image ( 360 A) with the blurred image ( 14 ) to determine a first blur difference between the first test image ( 360 A) and the blurred image ( 14 ); (iv) comparing the first perpendicular test image ( 366 A) with the blurred image ( 14 ) to determine a first perpendicular blur difference between the first perpendicular test image ( 366 A) and the blurred image ( 14 ); and (v) determining a first pair difference between the first blur difference and the first perpendicular blur difference.

BACKGROUND

Cameras are commonly used to capture an image of a scene that includesone or more objects. Unfortunately, some of the images are blurred. Forexample, movement of the camera and/or movement of the objects in thescene during the exposure time of the camera can cause motion blur inthe image that is mainly in the direction of motion.

There exist a number of deconvolution methods for reducing blur in ablurry image. These methods require a point spread function (“PSF”),which describes the blur, to be known or automatically estimated.Typically, the methods that estimate the PSF require a good initialguess for certain blur parameters, such as blur direction.

SUMMARY

The present invention is directed to a method and device for estimatinga blur direction of motion blur in a blurred image. In one embodiment,the method includes the steps of (i) blurring the blurred image along afirst test direction to create an artificially blurred first test image;(ii) blurring the blurred image along a first perpendicular testdirection to create an artificially blurred first perpendicular testimage, the first perpendicular test direction being perpendicular to thefirst test direction; (iii) comparing the first test image with theblurred image to determine a first blur difference between the firsttest image and the blurred image; (iv) comparing the first perpendiculartest image with the blurred image to determine a first perpendicularblur difference between the first perpendicular test image and theblurred image; and (v) determining a first pair difference between thefirst blur difference and the first perpendicular blur difference.

In certain embodiments, the proposed method for estimating the blurdirection is based on the concepts that (i) when artificial blur isapplied to the blurred image in a test direction that is similar to theblur direction, the difference in the image appearance is relativelysmall, and minimum changes exist between the additionally blurred imageand the original image; and (ii) when artificial blur is applied to theblurred image in a test direction that is perpendicular to the blurdirection, the difference in the image appearance is relatively large,and maximum changes exist between the additionally blurred image and theoriginal image.

Additionally, the method can include the steps of (i) blurring theblurred image along a second test direction to create an artificiallyblurred second test image, the second test direction being differentthan the first test direction; (ii) blurring the blurred image along asecond perpendicular test direction to create an artificially blurredsecond perpendicular test image, the second perpendicular test directionbeing perpendicular to the second test direction; (iii) comparing thesecond test image with the blurred image to determine a second blurdifference between the second test image and the blurred image; (iv)comparing the second perpendicular test image with the blurred image todetermine a second perpendicular blur difference between the secondperpendicular test image and the blurred image; and (v) determining asecond pair difference between the second blur difference and the secondperpendicular blur difference.

Moreover, the method can include the steps of (i) blurring the blurredimage along a third test direction to create an artificially blurredthird test image, the third test direction being different than thefirst test direction and the second test direction; (ii) blurring theblurred image along a third perpendicular test direction to create anartificially blurred third perpendicular test image, the thirdperpendicular test direction being perpendicular to the third testdirection; (iii) comparing the third test image with the blurred imageto determine a third blur difference between the third test image andthe blurred image; (iv) comparing the third perpendicular test imagewith the blurred image to determine a third perpendicular blurdifference between the third perpendicular test image and the blurredimage; and (v) determining a third pair difference between the thirdblur difference and the third perpendicular blur difference.

As provided herein, the method can include comparing one or more of thepair differences to select the blur direction. For example, the methodcan include the step of comparing the first pair difference, the secondpair difference, and the third pair different to estimate the blurdirection. More specifically, the method includes the step of selectingone of the first test directions as the blur direction in the event thefirst pair difference is greater than the second pair difference and thethird pair difference.

The present invention is also directed to a device for estimating a blurdirection of motion blur in a blurred image. In this embodiment, thecontrol system can perform some or all of the steps described above.

In yet another embodiment, the present invention is directed to a methodand device for deconvolving the blurred image.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of this invention, as well as the invention itself,both as to its structure and its operation, will be best understood fromthe accompanying drawings, taken in conjunction with the accompanyingdescription, in which similar reference characters refer to similarparts, and in which:

FIG. 1 is a simplified view of a scene, an image apparatus havingfeatures of the present invention, and a blurred image;

FIG. 2 is a simplified front perspective view of the image apparatus ofFIG. 1;

FIG. 3A illustrates a pair of artificially blurred first test images;

FIG. 3B illustrates a pair of artificially blurred second test images;

FIG. 3C illustrates a pair of artificially blurred third test images;

FIG. 3D illustrates a pair of artificially blurred fourth test images;

FIG. 4 illustrates the blurred image, and an adjusted image;

FIG. 5 is a flow chart that illustrates one method for estimating blurdirection; and

FIG. 6 illustrates another embodiment of a system having features of thepresent invention.

DESCRIPTION

FIG. 1 is a simplified perspective illustration of an image apparatus 10having features of the present invention, and a scene 12. The imageapparatus 10 captures a raw captured image 14 (illustrated away from theimage apparatus 10) that can be blurred 16 (illustrated as a thicker,wavy line). In one embodiment, the image apparatus 10 includes a controlsystem 18 (illustrated in phantom) that uses a unique method forestimating an unknown blur direction 20 (illustrated as an arrow) ofmotion blur 16 in the blurred image 14. In certain embodiments, withinformation regarding the blur direction 20 in the blurred image 14, theamount of blur 16 in the image 14 can be accurately reduced.

As an overview, in one embodiment, the proposed method for estimatingthe prevailing blur direction 20 of motion blur 16 is based on theconcepts that (i) when artificial blur is applied to the blurred image14 in a test direction that is similar to the blur direction 20, thedifference in the image appearance is relatively small, and minimumchanges exist between the additionally blurred image and the originalimage 14; and (ii) when artificial blur is applied to the blurred image14 in a test direction that is perpendicular to the blur direction 20,the difference in the image appearance is relatively large, and maximumchanges exist between the additionally blurred image and the originalimage 14.

The type of scene 12 captured by the image apparatus 10 can vary. Forexample, the scene 12 can include one or more objects 22, e.g. animals,plants, mammals, and/or environments. For simplicity, in FIG. 1, thescene 12 is illustrated as including one object 22. Alternatively, thescene 12 can include more than one object 22. In FIG. 1, the object 22is a simplified stick figure of a person.

It should be noted that movement of the image apparatus 10 and/ormovement of the object 22 in the scene 12 during the capturing of theblurred image 14 can cause motion blur 16 in the blurred image 14 thatis mainly in the blur direction 20. For example, in FIG. 1, the imageapparatus 10 was moved along a motion direction 24 (illustrated as anarrow) during the exposure time while capturing the blurred image 14. Asa result thereof, the blurred image 14 has blur 16 in the blur direction20 that corresponds to the motion direction 24.

It should be noted that the motion direction 24 is usually random andcan be different than that illustrated in FIG. 1. For example, themotion direction 24 can be up and down. This motion can be non-uniformlinear motion. Alternatively, the motion can be non-linear.

FIG. 2 illustrates a simplified, front perspective view of one,non-exclusive embodiment of the image apparatus 10. In this embodiment,the image apparatus 10 is a digital camera, and includes an apparatusframe 236, an optical assembly 238, and a capturing system 240(illustrated as a box in phantom), in addition to the control system 18(illustrated as a box in phantom). The design of these components can bevaried to suit the design requirements and type of image apparatus 10.Further, the image apparatus 10 could be designed without one or more ofthese components. Additionally or alternatively, the image apparatus 10can be designed to capture a video of the scene 12.

The apparatus frame 236 can be rigid and support at least some of theother components of the image apparatus 10. In one embodiment, theapparatus frame 236 includes a generally rectangular shaped hollow bodythat forms a cavity that receives and retains at least some of the othercomponents of the camera.

The apparatus frame 236 can include an aperture 244 and a shuttermechanism 246 that work together to control the amount of light thatreaches the capturing system 240. The shutter mechanism 246 can beactivated by a shutter button 248. The shutter mechanism 246 can includea pair of blinds (sometimes referred to as “blades”) that work inconjunction with each other to allow the light to be focused on thecapturing system 240 for a certain amount of time. Alternatively, forexample, the shutter mechanism 246 can be all electronic and contain nomoving parts. For example, an electronic capturing system 240 can have acapture time controlled electronically to emulate the functionality ofthe blinds.

The optical assembly 238 can include a single lens or a combination oflenses that work in conjunction with each other to focus light onto thecapturing system 240. In one embodiment, the image apparatus 10 includesan autofocus assembly (not shown) including one or more lens movers thatmove one or more lenses of the optical assembly 238 in or out until thesharpest possible image of the subject is received by the capturingsystem 240.

The capturing system 240 captures information for the raw blurred image14 (illustrated in FIG. 1). The design of the capturing system 240 canvary according to the type of image apparatus 10. For a digital typecamera, the capturing system 240 includes an image sensor 250(illustrated in phantom), a filter assembly 252 (illustrated inphantom), and a storage system 254 (illustrated in phantom).

The image sensor 250 receives the light that passes through the aperture244 and converts the light into electricity. One non-exclusive exampleof an image sensor 250 for digital cameras is known as a charge coupleddevice (“CCD”). An alternative image sensor 250 that may be employed indigital cameras uses complementary metal oxide semiconductor (“CMOS”)technology.

The image sensor 250, by itself, produces a grayscale image as it onlykeeps track of the total quantity of the light that strikes the surfaceof the image sensor 250. Accordingly, in order to produce a full colorimage, the filter assembly 252 is generally used to capture the colorsof the image.

The storage system 254 stores the various raw images 14 (illustrated inFIG. 1) and/or one or more adjusted images 455 (illustrated in FIG. 4)before these images are ultimately printed out, deleted, transferred ordownloaded to an auxiliary storage system or a printer. The storagesystem 254 can be fixedly or removable coupled to the apparatus frame236. Non-exclusive examples of suitable storage systems 254 includeflash memory, a floppy disk, a hard disk, or a writeable CD or DVD.

The control system 18 is electrically connected to and controls theoperation of the electrical components of the image apparatus 10. Thecontrol system 18 can include one or more processors and circuits, andthe control system 18 can be programmed to perform one or more of thefunctions described herein. In FIG. 2, the control system 18 is securedto the apparatus frame 236 and the rest of the components of the imageapparatus 10. Further, the control system 18 is positioned within theapparatus frame 236.

In certain embodiments, the control system 18 includes software thatestimates the blur direction 20 of motion blur 16 in the blurred image14. Further, the control system 18 can include software that reduces theblur 16 in the blurred image 14 to provide an adjusted image 455(illustrated in FIG. 4).

Referring back to FIG. 1, the image apparatus 10 includes an imagedisplay 56 that displays the blurred image 14 and/or the adjusted images455. With this design, the user can decide which images 14, 455 shouldbe stored and which images 14, 455 should be deleted. In FIG. 1, theimage display 56 is fixedly mounted to the rest of the image apparatus10. Alternatively, the image display 56 can be secured with a hingemounting system (not shown) that enables the display 56 to be pivoted.One non-exclusive example of an image display 56 includes an LCD screen.

Further, the image display 56 can display other information that can beused to control the functions of the image apparatus 10.

Moreover, the image apparatus 10 can include one or more controlswitches 58 electrically connected to the control system 18 that allowsthe user to control the functions of the image apparatus 10. Forexample, one or more of the control switches 58 can be used toselectively switch the image apparatus 10 to the blur direction 20estimation processes and/or deblurring processes disclosed herein.

As provided herein, in one embodiment, the motion blur of the capturedimage 14 (illustrated in FIG. 1) is assumed to be equal to H θ, where Hθ is a blur filter, H is a notation for a motion blurred filter, andtheta (“θ”) is the motion blur direction.

In one embodiment, the present invention applies additional blur, B α,in which B is a notation for another blur filter, and alpha (“α”) is themotion blur direction.

With this design, an overall blur (“Pall”) can be expressed as follows:

Pall=Bα*Hθ  Equation 1

As provided herein, if the direction alpha of the additional blurcoincides with the direction theta of the original blur, then theoverall blur will undergo minimum change. Basically, the blur shape willnot change, only the blur weight will change.

Alternatively, if the direction alpha of the additional blur isdifferent than the direction theta of the original blur, then theoverall blur will change in shape and blur weight. For example, if theadditional blur direction alpha is perpendicular to original blurdirection theta (i.e., α=θ+90), then the overall blur (“Pall”) willundergo maximum changes in shape and weight compared to other additionalblur directions.

In certain embodiments, the present invention works on the premise (i)that the correct estimated alpha will result in minimum changes betweenthe additionally blurred image to the original given blurred image, and(ii) the direction perpendicular to the correct estimation will resultin maximum changes between the additionally blurred image to theoriginal given blurred image.

This can be express in equations 2 and 3 below:

min α∥Bα|(x,y)−(x,y)∥  Equation 2

max α+90∥Bα+90|(x,y)−|(x,y)∥  Equation 3

where |(x,y) is the original given blurred image.

The present invention proposes to use both the minimum and maximuminformation to increase the accuracy of the motion blur directionestimation.

A blur difference “f(α)” between an artificially blurred image (blurredwith blur filter B and at an angle of alpha) and the original image canbe defined as follows:

f(α)=|Bα|(x,y)−|(x,y)|.  Equation 4

Further, a perpendicular blur difference “f(α+90)” between anartificially blurred image (blurred with blur filter B and at an angleof alpha plus ninety degrees) and the original image can be defined asfollows:

f(α+90)=|Bα+90 |(x,y)−|(x,y)|.  Equation 5

Moreover, a pair difference “PD” between the blur difference “f(α)” andthe perpendicular blur difference “f(α+90)” can be expressed as follows:

PD=|f(α+90)−f(α)|  Equation 6

As α moves away from the correct blur direction, f(α) will increaseaccordingly, and f(α+90) will decrease accordingly, therefore thedifference PD between them will be smaller and smaller.

The pair difference “|f(α+90)−f(α) |” should be a sharper curve than theblur difference “f(α)” as it roughly doubles the differences, thereforeit is easy and more robust to find the maximum. As a result thereof, thepresent method can lead to more accurate blur direction estimation andmore efficient implementation. Further, the present invention canrequire a very short testing blur size (e.g. length=3) to achieve goodresults because of the use of both the minimum and maximum information.This saves computation and allows for fast implementation.

The results maybe especially good if the original blur size iscomparably bigger than the test blur size.

To estimate the blur direction, the present invention selects aplurality of sample angles α that are in the range of zero to ninetydegrees [0 90]. Subsequently, (i) linear blur B is applied at eachsample angle α to the given blurred image |(x,y) to get B α |(x,y) foreach sample angle, and (ii) linear blur B is applied at each sampleangle α+90 to the given blurred image |(x,y) to get B α+90 |(x,y) foreach sample angle.

Next, the pair difference for each pair of angles (αα+90) is determinedto the maximum pair difference PD=|f(α+90)−f(α) |. Subsequently, afterthe maximum pair difference is determined, the estimated blur directionis: α, if f(α)<f(α+90) or a+90, if f(α+90)<f(α).

The present methods can better understood in conjunction with thediscussion of FIGS. 3A-3D. More specifically, FIG. 3A illustrates (i) anartificially blurred first test image 360A that is created by blurringthe blurred captured image 14 (illustrated in FIG. 1) along a first testdirection 362A (illustrated as an arrow); and (ii) an artificiallyblurred first perpendicular test image 364A that is created by blurringthe blurred captured image 14 along a first perpendicular test direction366A (illustrated as an arrow). In an orientation system 368 illustratedin FIG. 3A, the first test direction 362A is approximately zero (0)degrees, and the first perpendicular test direction 366A isapproximately ninety (90) degrees. Thus, the first perpendicular testdirection 366A is perpendicular to the first test direction 362A. Thefirst test image 360A and the first perpendicular test image 364A can becollectively referred to as a pair of first test images or a first imagepair.

It should be noted that the terms “first”, “second”, “third”, and“fourth” are used merely for convenience and that any of the images canbe called the “first”, “second”, “third”, or “fourth”.

Somewhat similarly, FIG. 3B illustrates (i) an artificially blurredsecond test image 360B that is created by blurring the blurred capturedimage 14 (illustrated in FIG. 1) along a second test direction 362B(illustrated as an arrow); and (ii) an artificially blurred secondperpendicular test image 364B that is created by blurring the blurredcaptured image 14 along a second perpendicular test direction 366B(illustrated as an arrow). In the orientation system 368 illustrated inFIG. 3B, the second test direction 362B is approximately twenty-five(25) degrees, and the second perpendicular test direction 366B isapproximately one hundred and fifteen (115) degrees. Thus, the secondperpendicular test direction 366B is perpendicular to the second testdirection 362B. The second test image 360B and the second perpendiculartest image 364B can be collectively referred to as a pair of second testimages or a second image pair.

FIG. 3C illustrates (i) an artificially blurred third test image 360Cthat is created by blurring the blurred captured image 14 (illustratedin FIG. 1) along a third test direction 362C (illustrated as an arrow);and (ii) an artificially blurred third perpendicular test image 364Cthat is created by blurring the blurred captured image 14 along a thirdperpendicular test direction 366C (illustrated as an arrow). In theorientation system 368 illustrated in FIG. 3C, the third test direction362C is approximately fifty (50) degrees, and the third perpendiculartest direction 366C is approximately one hundred and forty (140)degrees. Thus, the third perpendicular test direction 366C isperpendicular to the third test direction 362C. The third test image360C and the third perpendicular test image 364C can be collectivelyreferred to as a pair of third test images or a third image pair.

FIG. 3D illustrates (i) an artificially blurred fourth test image 360Dthat is created by blurring the blurred captured image 14 (illustratedin FIG. 1) along a fourth test direction 362D (illustrated as an arrow);and (ii) an artificially blurred fourth perpendicular test image 364Dthat is created by blurring the blurred captured image 14 along a fourthperpendicular test direction 366D (illustrated as an arrow). In theorientation system 368 illustrated in FIG. 3D, the fourth test direction362D is approximately seventy-five (75) degrees, and the fourthperpendicular test direction 366D is approximately one hundred andsixty-five (165) degrees. Thus, the fourth perpendicular test direction366D is perpendicular to the fourth test direction 362D. The fourth testimage 360A and the fourth perpendicular test image 364A can becollectively referred to as a pair of fourth test images or a fourthimage pair.

Further, FIGS. 3A-3D includes the actual blur direction 20 (illustratedas an arrow).

In one embodiment, each of the test images 360A-D, 364A-D is generatedby artificially blurring the captured image 14 in the respective testdirection 362A-D, 366A-D. For example, to generate the first test image360A, a convolution operation is performed on the blurred image 14 witha matrix representing Point Spread Function (“PSF”) corresponding toblurring in the first test direction 362A (horizontal direction). Eachof the test images 360A-D, 364A-D can be generated using the convolutionoperation in a somewhat similar fashion.

In FIGS. 3A-3D, the original blur 16 is again illustrated with thethicker, wavy line. Further, in FIGS. 3A-3D, each of the test images360A-D, 362A-D includes an additional artificial blur 370 represented as“B's”.

As provided herein, when more blur is applied to the blurred image 14 ina test direction 362A-D, 366A-D that is similar to the original blurdirection 20, the difference in the image appearance is relatively smalland the amount of additional artificial blur 370 is relatively small.However, when more blur is applied to the blurred image 14 in a testdirection 362A-D, 366A-D that is very different (e.g. perpendicular) tothe blur direction 20, the difference in the image appearance isrelatively large and the amount of additional artificial blur 370 isrelatively large.

As the test direction 362A-D, 366A-D moves away from the original blurdirection 20, the amount of blur will increase accordingly until thetest direction 362A-D, 366A-D is approximately perpendicular to theoriginal blur direction 20. In the example illustrated in FIGS. 3A-3D,the original blur direction 20 is approximately at one hundred and forty(140) degrees relative to the orientation system 368. Further, in FIGS.3A-3D, the third perpendicular test image 364C has the least amount ofadditional blurring 370 while the third test image 360C has the largestamount of additional blurring 370. This is because the thirdperpendicular test direction 366C is equal to the original blurdirection 20, and the third test direction 362C is perpendicular to theoriginal blur direction 20.

As provided herein, the control system 18 (illustrated in FIG. 1)computes a pair difference for each image pair. For example, to computethe pair difference for the first image pair, the control system 18 (i)compares the first test image 360A with the blurred image 14 todetermine a first blur difference between the first test image 360A andthe blurred image 14; (ii) compares the first perpendicular test image364A with the blurred image 14 to determine a first perpendicular blurdifference between the first perpendicular test image 364A and theblurred image 14. Next, the control system 18 calculates a first pairdifference between the first blur difference and the first perpendicularblur difference.

Similarly, to compute the pair difference for the second image pair, thecontrol system 18 (i) compares the second test image 360B with theblurred image 14 to determine a second blur difference between thesecond test image 360B and the blurred image 14; (ii) compares thesecond perpendicular test image 364B with the blurred image 14 todetermine a second perpendicular blur difference between the secondperpendicular test image 364B and the blurred image 14. Next, thecontrol system 18 calculates a second pair difference between the secondblur difference and the second perpendicular blur difference.

Further, to compute the pair difference for the third image pair, thecontrol system 18 (i) compares the third test image 360C with theblurred image 14 to determine a third blur difference between the thirdtest image 360C and the blurred image 14; (ii) compares the thirdperpendicular test image 364C with the blurred image 14 to determine athird perpendicular blur difference between the third perpendicular testimage 364C and the blurred image 14. Next, the control system 18calculates a third pair difference between the third blur difference andthe third perpendicular blur difference.

Moreover, to compute the pair difference for the fourth image pair, thecontrol system 18 (i) compares the fourth test image 360D with theblurred image 14 to determine a fourth blur difference between thefourth test image 360D and the blurred image 14; (ii) compares thefourth perpendicular test image 364D with the blurred image 14 todetermine a fourth perpendicular blur difference between the fourthperpendicular test image 364D and the blurred image 14. Next, thecontrol system 18 calculates a fourth pair difference between the fourthblur difference and the fourth perpendicular blur difference.

Further, the control system 18 compares the pair differences for theimage pairs and selects the pair difference with the largest value.Subsequently, for the image pair with the largest pair difference, thecontrol system 18 selects the test direction with the smallest blurdifference as the estimated blur direction. In the example illustratedin FIGS. 3A-3D, the third pair difference is the largest because thethird perpendicular test image 364C has the least amount of additionalblurring 370 between the while the third test image 360C has the largestamount of additional blurring 370. Stated in another fashion, the thirdpair difference is the largest because, the third perpendicular blurdifference is relatively small while the third blur difference isrelatively large.

After the third test pair is selected by the control system 18, thethird perpendicular test direction 366C is selected by the controlsystem 18 as the unknown blur direction because the third perpendicularblur difference is less than the third blur difference.

It should be noted that the difference between what is considered alarge blur difference and what is considered a small blur differencewill vary according to the content of the image and many other factors,such as size of the image. Also, there is a number of different ways howto measure the difference between two images. The resulting value can bepractically any number or designation that can be used to compare thevalues for the different directions in the same image.

The number of test image pairs used in the estimation and the differencebetween the test directions 362A-D can vary pursuant to the teachingsprovided herein. Generally speaking, the accuracy of the estimation canincrease as the number of image pairs is increased, but thecomputational complexity also increases as the number of image pairscreated is increased.

In FIGS. 3A-3D, only four image pairs are provided for simplicity andthe test directions 362A-D, 366A-D are oriented approximatelytwenty-five degrees apart. In alternative non-exclusive embodiments,ten, twenty, forty-five, ninety, or one hundred and eighty test imagescan be generated, and the test directions can be spaced apartapproximately eighteen, nine, six, four, two, or one degrees.

Alternatively, a coarse-to-fine approach in the sampling angles can beused. In this example, coarse sampling (e.g. every ten degree) is usedby the control system to obtain the rough direction. Subsequently, nearthe rough direction, dense sampling (e.g. every one degree) is used bythe control system to obtain the fine estimated blur direction.

In certain embodiments, the present invention can be applied to either amonochrome image or a color image (convert color to grayscale). Also, asprovided herein, the blur direction estimation can be applied toprocessed images, or the blur direction estimation can also beimplemented as a part of image processing pipeline.

In one non-exclusive example, for a color image, the blur differencescan be calculated with the control system 18 (illustrated in FIG. 1) bycomparing the brightness value at each pixel in each channel matrix forthe blurred image 14 to the brightness value at each pixel in eachchannel matrix in the respective test image. In this example, the blurdifference can be calculated for each channel, and values averaged topossibly get a more robust blur direction estimate. However, this methodcan be computationally very expensive.

In another example, a color image would first be converted to black andwhite, for example by taking the average of the three color channels, orby selecting one of the channels (usually the green one is used). Next,the method is applied to the resulting black and white image.

Alternatively or additionally, one or more of the blur difference valuescan be generated by interpolation information from previously generatedblur difference values for test images that were generated using theconvolution operation. In one non-exclusive embodiment, test images aregenerated at five degree intervals using the convolution method.Subsequently, additional blur difference values can be generated at onedegree increments between the previously generated blur differencevalues for the test images using interpolation.

FIG. 4 illustrates the blurred image 14, and the adjusted image 455. Inthis embodiment, after the blur direction 20 is estimated, the controlsystem 18 (illustrated in FIG. 1) can perform one or more deblurringtechniques to target the blur 16 in the blurred image 14 to provide theadjusted image 455. For example, accelerated Lucy-Richardsondeconvolution can be performed on the blurred image 14 to provide theadjusted image 455. In this example, the adjusted image 455 hassignificantly less blur 16 than the capture image 14.

To deblur an image, you have to know the PSF (which is the function thatdescribes how the image is blurred). In case of a motion blur, anassumption is often made that the motion is uniform linear motion (inpractice it works only for relatively small blurs, though). In thatcase, to find the PSF you need to estimate blur direction and blurlength. The present invention deals with determining the direction ofmotion blur. A separate method may be necessary to estimate blur length.

So called “blind deconvolution methods” assume that the PSF is unknownand they attempt both to find PSF and to produce a deblurred image atthe same time. These methods are typically iterative methods, theyrequire some initial guess for PSF, and this initial guess needs to beclose enough to the real PSF for the method to be successful. Knowingthe blur direction can help to generate a good initial guess.

FIG. 5 is a flow chart that illustrates one method for estimating adirection of motion blur. First a step 510, the motion blurred image iscaptured by the camera. Subsequently, at step 512, the control systemgenerates a plurality of artificially blurred image pairs. Next, at step514, the control system compares each of the artificially blurred imagesto the original blurred image to generate a blur difference for eachartificially blurred image. Subsequently, at step 516, the controlsystem compares the blur differences for each image pair to determine apair difference for each image pair. Next, at step 518, the controlsystem selects the image pair with the greatest pair difference as theimage pair that includes the estimated blur direction. Subsequently, atstep 520, the control system compares the blur differences for theselected image pair, and the control system identifies the blurdirection of the selected image pair that has the lowest blurdifference. Finally, at step 522, the control system deblurs theoriginal captured image to generate the adjusted image.

FIG. 6 illustrates another embodiment of an estimating system 672 havingfeatures of the present invention. In this embodiment, the imageapparatus 10 again captures the blurred image 14 (illustrated in FIG.1). However, in this embodiment, the blurred image 14 is transferred toa computer 674 (e.g. a personal computer) that includes a computercontrol system 618 (illustrated in phantom) that uses the estimationmethod disclosed herein to estimate the blur direction. Further, thecomputer control system 618 can deblur the blurred image 14 and providethe adjusted image 455 (illustrated in FIG. 4).

While the current invention is disclosed in detail herein, it is to beunderstood that it is merely illustrative of the presently preferredembodiments of the invention and that no limitations are intended to thedetails of construction or design herein shown other than as describedin the appended claims.

1. A method for estimating a blur direction of motion blur in a blurredimage, the method comprising the steps of: blurring the blurred imagealong a first test direction to create an artificially blurred firsttest image; blurring the blurred image along a first perpendicular testdirection to create an artificially blurred first perpendicular testimage, the first perpendicular test direction being substantiallyperpendicular to the first test direction; comparing the first testimage with the blurred image to determine a first blur differencebetween the first test image and the blurred image; comparing the firstperpendicular test image with the blurred image to determine a firstperpendicular blur difference between the first perpendicular test imageand the blurred image; and determining a first pair difference betweenthe first blur difference and the first perpendicular blur difference.2. The method of claim 1 further comprising the steps of: blurring theblurred image along a second test direction to create an artificiallyblurred second test image, the second test direction being differentthan the first test direction; blurring the blurred image along a secondperpendicular test direction to create an artificially blurred secondperpendicular test image, the second perpendicular test direction beingsubstantially perpendicular to the second test direction; comparing thesecond test image with the blurred image to determine a second blurdifference between the second test image and the blurred image;comparing the second perpendicular test image with the blurred image todetermine a second perpendicular blur difference between the secondperpendicular test image and the blurred image; and determining a secondpair difference between the second blur difference and the secondperpendicular blur difference.
 3. The method of claim 2 furthercomprising the step of comparing the first pair difference with thesecond pair difference to estimate the blur direction.
 4. The method ofclaim 3 further comprising the step of selecting one of the first testdirections as the blur direction in the event the first pair differenceis greater than the second pair difference.
 5. The method of claim 2further comprising the steps of: blurring the blurred image along athird test direction to create an artificially blurred third test image,the third test direction being different than the first test directionand the second test direction; blurring the blurred image along a thirdperpendicular test direction to create an artificially blurred thirdperpendicular test image, the third perpendicular test direction beingsubstantially perpendicular to the third test direction; comparing thethird test image with the blurred image to determine a third blurdifference between the third test image and the blurred image; comparingthe third perpendicular test image with the blurred image to determine athird perpendicular blur difference between the third perpendicular testimage and the blurred image; and determining a third pair differencebetween the third blur difference and the third perpendicular blurdifference.
 6. The method of claim 5 further comprising the step ofcomparing the first pair difference with the second pair difference andthe third pair different to estimate the blur direction.
 7. The methodof claim 6 further comprising the step of selecting one of the firsttest directions as the blur direction in the event the first pairdifference is greater than the second pair difference and the third pairdifference.
 8. The method of claim 1 further comprising the step ofdeconvolving the blurred image to provide an adjusted image.
 9. A devicefor estimating a blur direction of motion blur in a blurred image, thedevice comprising: a control system that (i) blurs the blurred imagealong a first test direction to create an artificially blurred firsttest image; (ii) blurs the blurred image along a first perpendiculartest direction to create an artificially blurred first perpendiculartest image, the first perpendicular test direction being substantiallyperpendicular to the first test direction; (iii) compares the first testimage with the blurred image to determine a first blur differencebetween the first test image and the blurred image; (iv) compares thefirst perpendicular test image with the blurred image to determine afirst perpendicular blur difference between the first perpendicular testimage and the blurred image; and (v) determines a first pair differencebetween the first blur difference and the first perpendicular blurdifference.
 10. The device of claim 9 wherein the control system (i)blurs the blurred image along a second test direction to create anartificially blurred second test image, the second test direction beingdifferent than the first test direction; (ii) blurs the blurred imagealong a second perpendicular test direction to create an artificiallyblurred second perpendicular test image, the second perpendicular testdirection being substantially perpendicular to the second testdirection; (iii) compares the second test image with the blurred imageto determine a second blur difference between the second test image andthe blurred image; (iv) compares the second perpendicular test imagewith the blurred image to determine a second perpendicular blurdifference between the second perpendicular test image and the blurredimage; and (v) determines a second pair difference between the secondblur difference and the second perpendicular blur difference.
 11. Thedevice of claim 10 wherein the control system compares the first pairdifference with the second pair difference to estimate the blurdirection.
 12. The device of claim 11 wherein the control system selectsone of the first test directions as the blur direction in the event thefirst pair difference is greater than the second pair difference. 13.The device of claim 10 wherein the control system (i) blurs the blurredimage along a third test direction to create an artificially blurredthird test image, the third test direction being different than thefirst test direction and the second test direction; (ii) blurs theblurred image along a third perpendicular test direction to create anartificially blurred third perpendicular test image, the thirdperpendicular test direction being substantially perpendicular to thethird test direction; (iii) compares the third test image with theblurred image to determine a third blur difference between the thirdtest image and the blurred image; (iv) compares the third perpendiculartest image with the blurred image to determine a third perpendicularblur difference between the third perpendicular test image and theblurred image; and (v) determines a third pair difference between thethird blur difference and the third perpendicular blur difference. 14.The device of claim 13 wherein the control system compares the firstpair difference with the second pair difference and the third pairdifferent to estimate the blur direction.
 15. The device of claim 14wherein the control system selects one of the first test directions asthe blur direction in the event the first pair difference is greaterthan the second pair difference and the third pair difference.
 16. Thedevice of claim 9 wherein the control system deconvolves the blurredimage to provide an adjusted image.
 17. The device of claim 9 furthercomprising a capturing system for capturing the blurred image.
 18. Amethod for estimating a blur direction of motion blur in a blurredimage, the method comprising the steps of: creating a first pair ofartificially blurred images by blurring the blurred image along a firsttest direction and along a first perpendicular test direction that issubstantially perpendicular to the first test direction; and determininga first pair difference between the first pair of artificially blurredimages.
 19. The method of claim 18 wherein the step of determining afirst pair difference includes the steps of (i) comparing eachartificially blurred images to the blurred image to create a blurdifference for each artificially blurred image; and (ii) comparing theblur differences for the artificially blurred images.
 20. The method ofclaim 18 further comprising the step of (i) creating a second pair ofartificially blurred images by blurring the blurred image along a secondtest direction and along a second perpendicular test direction that issubstantially perpendicular to the second test direction; and (ii)determining a second pair difference between the second pair ofartificially blurred images.
 21. The method of claim 20 furthercomprising the step of comparing the first pair difference with thesecond pair difference to estimate the blur direction.
 22. The method ofclaim 21 further comprising the step of selecting one of the first testdirections as the blur direction in the event the first pair differenceis greater than the second pair difference.